Hypothesis testing: Errors, Power, Effect size, and Assumptions

class: center, middle, inverse, title-slide

.title[
# Hypothesis testing: Errors, Power, Effect size, and Assumptions
]
.subtitle[
## Data Analysis for Psychology in R 1 Semester 2, Week 5
]
.author[
### Dr Umberto Noè
]
.institute[
### Department of Psychology The University of Edinburgh
]

---

# Learning objectives

1. Understand what are Type I and Type II errors in hypothesis testing.

1. Recognise the significance level as measuring the tolerable chance of committing a Type I error.

1. Recognise the effect of sample size on power.

1. Be able to check the assumptions underlying the t-test for a population mean.

---
class: inverse, center, middle

# Part A
## Errors and Power

---
# Where we're going to

- Hypothesis testing lets us determine whether, for example, an observed difference between a sample mean and an hypothesised value is real or just due to random sampling variation.

- However, statistical significance sometimes may lead us to wrong conclusions!

- It is possible to make two kinds of wrong decisions:

- rejecting a true null hypothesis 
  
--

- not rejecting a false null hypothesis

- This week we will discuss common pitfalls of hypothesis testing, as well as the factors that influence the probability of committing these errors.

---
# Errors in hypothesis testing

- Whether your decision is either to (a) reject the null hypothesis or (b) not reject the null hypothesis, you might be making an error.

- The reasoning of hypothesis tests is often compared to that of a court trial. The possibilities in such a trial are given below.

- Possible outcomes of testing `$H_0 : \text{Defendant = Not guilty}$`:

---
# Errors in hypothesis testing

- In our system of justice, convicting an innocent person is considered worse than letting a guilty person go.

- Possible outcomes of testing `$H_0 : \text{Defendant = Not guilty}$`:

---
# Errors in hypothesis testing

- Similarly, there are two types of errors in hypothesis testing.

.pull-left[
__Type I error__

- You could convict an innocent

- i.e., you could reject "not guilty" when the person is truly "not guilty"

- i.e., you could reject the null hypothesis when it is true
]

.pull-right[
__Type II error__

- You could fail to convict a guilty defendant

- i.e., you could fail to reject "not guilty" when the defendant is guilty

- i.e., you could fail to reject the null hypothesis when it is false
]

- Like convicting an innocent person, the error of rejecting a true null hypothesis is considered more serious, and so a null hypothesis isn't rejected unless the evidence against it is convincing beyond reasonable doubt.

---
# Errors in hypothesis testing

- Whether your decision is either to reject the null hypothesis or to not reject the null hypothesis, you might be making an error.

+ You make a **Type I error** when you reject a true null hypothesis

+ You make a **Type II error** when you don't reject a false null hypothesis

- Possible outcomes of testing `$H_0 : \mu = \mu_0$`

---
# When is a Type II error worse?

- Sometimes, we are more worried about committing a Type II error than a Type I error.

+ This is application-specific, but it happens less often.
    
    + You should think about your particular study to see if this is the case.
    
- Example: developing a rapid test for diabetes:

`$$H_0 : \text{Person} = \text{Non-diabetic}$$`

---
# When is a Type II error worse?

- Null hypothesis: Patient is not diabetic

- Alternative hypothesis: Patient is diabetic

- __Type I error__ = __False Positive__:

+ a test that indicates a patient has diabetes when in reality they don't.

- __Type II error__ = __False Negative__:

+ A test that indicates the patient does not have diabetes when in fact they do have it. That is, a test that fails to detect an actual diabetic.

+ More of a concern, as a person will miss out on treatment.